The present disclosure relates to an image processing apparatus, an image processing method, and a program. In particular, the present disclosure relates to an image processing apparatus, an image processing method, and a program that generate images with a high dynamic range (wide dynamic range).
A solid-state image pickup device such as a CCD image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor used in a video camera or a digital still camera carries out photoelectric conversion by accumulating charge in keeping with the amount of incident light and outputting an electrical signal corresponding to the accumulated charge. However, there is a limit on the amount of charge that can be accumulated in a photoelectric conversion element, so that when a certain amount of light has been received, a saturation level is reached, resulting in regions of a subject with a certain brightness or higher being set at a saturation luminance level, a problem referred to as “blown out highlights” or “clipping”.
To prevent clipping, processing is carried out to control the charge accumulation period of the photoelectric conversion element in accordance with the change in light from outside or the like to adjust the exposure length and thereby optimize sensitivity. For example, by using a high shutter speed to shorten the exposure length for a bright subject, the charge accumulation period of the photoelectric conversion element is reduced and an electrical signal is outputted before the amount of accumulated charge reaches the saturation level. By carrying out such processing, it is possible to output an image in which tones are correctly reproduced for the subject.
However, if a high shutter speed is used when photographing a subject in which both bright and dark regions are present, the exposure length will not be sufficient for the dark regions, which will result in deterioration in the S/N ratio and a fall in image quality. To correctly reproduce the luminance levels of bright regions and dark regions in a photographed image of a subject that includes both bright and dark regions, it is necessary to use a long exposure for pixels on the image sensor where there is little incident light to achieve a high S/N ratio and to carry out processing to avoid saturation for pixels with large amounts of incident light.
One known method of realizing such processing is to consecutively pick up a plurality of images with different periods of exposure time and then combine such images. That is, a long-time exposure image and a short-time exposure image are separately and consecutively picked up and a combining process that uses the long-time exposure image for dark image regions and the short-time exposure image for bright image regions where clipping would occur for the long-time exposure image is carried out to produce a single image. In this way, by combining the plurality of images with different exposures, it is possible to produce images with a high dynamic range with no clipping, that is, images with a wide dynamic range (HDR images).
For example, Japanese Patent Laid-Open Publication No. 2000-50151 discloses a configuration for producing a wide dynamic range image by consecutively picking up two images with different periods of exposure time and then combining such images. With reference to FIG. 1, such processing is described. An image pickup device outputs two images with different periods of exposure time within a video rate (30-60 fps), for example, in shooting videos. In addition, two images with different periods of exposure time are produced and output also in photographing still images. FIG. 1 depicts features of images (long-time exposure image and short-time exposure image) with different periods of exposure time produced by the image pickup device. A horizontal axis is a time (t) and a vertical axis is an accumulated charge amount (e) in a light receiving photo diode (PD) that composes a photoelectric conversion element corresponding to one pixel of a solid-state image pickup element.
For example, when the light receiving photo diode (PD) receives much light, that is, in the case of a bright subject, the accumulated charge amount sharply increases along with time elapsed, as shown in a high luminance region 11 shown in FIG. 1. Meanwhile, when the light receiving photo diode (PD) receives less light, that is, in the case of a dark subject, the accumulated charge amount gently increases along with time elapsed, as shown in a low luminance region 12 shown in FIG. 1.
Times t0 to t3 correspond to exposure times TL to produce long-time exposure images. The line shown by the low luminance region 12 as such long-time exposure times TL can obtain accurate tones by the tone level of a pixel determined using electrical signals obtained based on such an accumulated charge amount (Sa) without reaching a saturation level (non-saturation point Py).
However, the line shown by a high luminance region 11 represents that before the time t3, the accumulated charge had already reached a saturation level (saturation point Px). Accordingly, such a high luminance region 11 obtains only a pixel value corresponding to an electrical signal of the saturation level from the long-time exposure images and becomes a clipping pixel as a result.
In such a high luminance region 11, the accumulated charges of the light receiving photo diode (PD) are swept once before reaching the time t3, for example, in the time t1 (charge sweeping start point P1) shown in FIG. 1. The charge sweeping is carried out to an intermediate voltage maintaining level controlled in the photo diode (PD), rather than with all of the charges accumulated in the light receiving photo diode (PD). After the charge sweeping, a short-time exposure is carried out for exposure times (t1 to t2), again. That is, the short-time exposure from a short-time exposure start point P2 to a short-time exposure end point P3, as shown in the figure, is carried out. An accumulated charge amount (Sb) is obtained through such a short-time exposure, and the tone level of a pixel is determined based on electrical signals obtained based on the accumulated charge amount (Sb).
Further, when a pixel value based on the electrical signals which are based on the accumulated charge amount (Sa) obtained through the long-time exposure in the low luminance region 12 and the electrical signals which are based on the accumulated charge amount (Sb) obtained through the short-time exposure in the high luminance region 11 is determined, an estimated accumulated charge amount when the exposures have been carried out for the same time or an electrical signal output value corresponding to the estimated accumulated charge amount is calculated, and a pixel level is determined based on the calculated result.
In this way, it is possible to produce images with a high dynamic range without clipping by combining the short-time exposure images and the long-time exposure images.
However, in the configuration described in Japanese Patent Laid-Open Publication No. 2000-50151, it is necessary to perform the process in which the long-time exposure images and the short-time exposure images are individually photographed and combined.
In this way, it is possible to produce the wide dynamic range images (HDR images) using the plurality of images with different periods of exposure time, but, for example, the following problems occur in processing based on the plurality of images.
Problem 1: It is necessary to photograph images several times, and in addition, it is necessary to provide with a memory to store these images.
Problem 2: Since a plurality of images with different photographing timings is combined or photographed data obtained through the long-time exposure is used, it is vulnerable to shaking of a camera.
In addition, image pickup elements used in many cameras have configurations in which color filters with, for example, an RGB array are mounted and light of a specific wavelength enters each pixel.
In particular, a color filter with, for example, a Bayer array is frequently used.
A picked-up image with a Bayer array becomes a mosaic image where only pixel values corresponding to any of RGB colors are set to each pixel of the picked-up image. The signal processing unit of a camera performs various signal processing such as pixel value interpolation for this mosaic image, performs de-mosaic processing of setting all pixel values of RGB to each pixel, and produces and outputs camera images.
It can be said that many studies have already been made for signal processing on a picked-up image with a color filter according to such a Bayer array, which has thus been technically established to some extent. However, it is also true that studies on signal processing on an image with an array different from the Bayer array are yet insufficient.
Further, for example, in addition to each color of RGB as a filter pertaining to a pickup image element, Japanese Patent Laid-Open Publication No. 2011-55038 discloses correction processing for a picked-up image of an image pickup apparatus with a filter with an RGBW array that has an entire wavelength transparent W (White) pixel.